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PREFACE 


This  paper  documents  work  performed  during  Phase  2  of  a  three-phase  project  entitled, 
“Desktop  Decision  Training  for  Logistics  Command  and  Control  (DDT/LC2).”  This  project  is 
being  accomplished  under  Contract  No.  F33615-91-C-0007,  with  Systems  Engineering 
Associates  (SEA),  San  Diego,  CA.  Management  of  this  effort  is  provided  by  the  Armstrong 
Laboratory,  Human  Resources  Directorate,  Technical  Training  Research  Division,  Instructional 
Systems  Branch  (AL/HRTD). 

This  project  is  being  accomplished  in  three  phases.  Phase  1  was  devoted  to  developing  a 
theory-based  instructional  strategy  and  a  PC-based  software  system  architecture  (Brecke  & 
Garcia,  1995).  Phase  2  focused  on  developing  an  initial  training  prototype.  Phase  3  is  oriented 
toward  further  development  of  the  prototype  into  a  deployable  training  system  that  can  also  serve 
as  a  research  vehicle  for  exploring  instructional  strategy  variations. 

This  paper  describes  work  performed  during  Phase  2  to  develop  the  initial  decision¬ 
making  training  prototype.  It  is  described  from  the  viewpoint  of  instructional  design  and 
documents  some  of  the  significant  and  interesting  problems  that  had  to  be  resolved.  A 
companion  report  (Van  de  Wetering  &  Garcia,  1995,  in  press)  describes  the  prototype  from  the 
software  engineering  perspective. 
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SUMMARY 


In  May  1989,  the  Air  Force  Logistics  Plans  and  Programs  Directorate  (HQ  USAF/LGX) 
requested  that  the  Air  Force  Human  Resources  Directorate  (HR)  develop  an  improved  training 
technology  for  Logistics  Command  and  Control  Centers  through  the  United  States  Air  Force 
(USAF).  The  objective  was  to  provide  a  means  of  training  logistics  decision  makers  to  work 
with  critical  information  and  achieve  the  best  use  of  resources.  This  tasking  originated  under  a 
Memorandum  of  Agreement  between  HQ  USAF/LGX,  Air  Force  Systems  Command  (now  the 
Air  Force  Material  Command),  and  HR. 

In  response,  HR  let  a  contract  with  Systems  Engineering  Associates  (SEA)  to  produce  a 
desktop  decision  trainer  which  will  provide  individual  instruction  and  enable  students  to  practice 
solving  realistic  logistics  problems  in  a  Logistics  Readiness  Center  environment.  The  project, 
which  began  in  February  1992,  will  be  completed  in  February  1997. 
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I.  INTRODUCTION 


The  objectives  of  the  “Desktop  Decision  Training  for  Logistics  Command  and  Control 
(DDT/LC2)”  research  and  development  effort  are  to:  (1)  identify  and  develop  instructional 
strategies  for  training  decision-making  skills;  (2)  identify  and  develop  decision-making  models 
to  serve  as  a  basis  for  training;  and  (3)  develop  an  experimental  computer-based  training 
prototype  that  combines  decision-making  lessons  with  a  simulation  environment  to  enable 
logistics  personnel  to  experience  similar  problems  and  situations  encountered  in  the  operational 
environment.  The  prototype  will  enable  students  to  practice  decision-making  skills  and  obtain 
feedback  relative  to  their  performance. 

This  project  is  being  accomplished  in  three  phases.  Phase  1  was  devoted  to  developing  a 
theory-based  instructional  strategy  and  PC-based  software  system  architecture  (Brecke  &  Garcia, 
1995).  Phase  2  focused  on  developing  an  initial  training  prototype.  Phase  3  is  oriented  toward 
further  development  of  the  prototype  into  a  deployable  training  system  that  can  also  serve  as  a 
research  vehicle  for  exploring  instructional  strategy  variations. 

This  paper  describes  work  performed  during  Phase  2  to  develop  the  initial  decision¬ 
making  training  prototype.  It  is  described  from  the  viewpoint  of  instructional  design  and 
documents  some  of  the  problems  that  needed  to  be  resolved.  A  companion  report  (Van  de 
Wetering  &  Garcia,  1995,  in  press)  describes  the  prototype  from  the  software  engineering 
perspective. 


II.  PHASE  2  PROTOTYPE  INSTRUCTIONAL  STRATEGY 


The  Phase  2  prototype  (also  referred  to  as  DDT  Beta  Version  0.1),  can  be  illustrated  by 
two  complementary  views  of  the  overall  system  design.  The  first  view  represents  the  DDT/LC2 
system  as  an  instructional  strategy  (Figure  1).  The  second  views  the  system  as  a  system 
"architecture"  of  software  components  (Figure  2). 

Figure  1.  Multi-level  instructional  strategy. 


Figure  1  represents  a  macro-view  of  the  instructional  strategy  for  the  desktop  decision 
trainer.  A  "course"  of  instruction  leading  to  the  desired  training  outcome  is  subdivided  into 
levels  of  elaboration  (Reigeluth,  1983).  The  first  level  is  the  "epitome"  level,  at  which  the 
student  is  exposed  to  the  simplest  possible  form  of  decision  making  in  a  chosen  domain.  For 
purposes  of  this  project,  the  domain  is  Air  Force  Logistics  Supply.  At  subsequent  levels,  the 
student  experiences  increasingly  elaborate  decision  making  situations  until  some  predetermined 
target  level  of  complexity  is  reached. 

Each  level  contains  two  different  forms  of  instruction  -  lessons  and  exercises.  Lessons 
precede  exercises  and  are  designed  as  standard  Computer  Assisted  Instruction  (CAI)  to  foster  the 
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acquisition  of  declarative  knowledge  about  how  variables  in  the  decision  making  situation 
influence  decision  goals  and  options.  Following  the  lessons,  the  student  is  subjected  to 
instruction  in  the  form  of  simulation  exercises.  This  form  of  instruction  no  longer  provides  well 
ordered  and  carefully  structured  presentations  with  content,  examples,  explanations,  and  practice, 
but  immerses  the  student  in  a  realistic  scenario  in  which  they  must  make  decisions  without 
further  guidance.  These  "exercises"  are  designed  to  foster  the  development  of  the  recognition- 
primed  decision  mode  (Klein  and  Calderwood  ,1990).  Once  the  student  achieves  a  specified 
level  of  reliability  in  that  mode,  they  proceed  to  the  next  level.  At  present,  The  DDT  Beta 
Version  0.1  includes  lessons  and  exercises  for  the  “epitome”  level. 

Figure  2.  System  architecture. 
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Figure  2  illustrates  the  software  architecture  used  to  implement  lessons  and  exercises. 
“Ovals”  represent  processes  and  programs  in  this  diagram,  while  “boxes”  represent  databases. 
The  system's  instructional  atom  database  includes  all  of  its  instructional  content.  This  database 
is  made  up  of  instructional  atoms,  which  are  individual  lessons  and  exercises.  As  suggested  in 
Figure  2,  instructional  atoms  are  created  by  authoring  applications  and  isplayed  by  presentation 
applications.  This  architecture  introduces  minimal  constraints  upon  the  content  of  lessons  and 
exercises  (instructional  atoms).  It  permits  any  kind  of  instruction  to  be  used,  provided  the 
instruction  is  accompanied  by  an  authoring  program  and  a  presentation  program. 
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The  outline  database  contains  instructional  strategies  for  courses.  Each  strategy  identifies 
the  instructional  atoms  that  will  be  presented  in  a  course  and  constrains  their  presentation.  For 
example,  an  instructional  strategy  that  includes  minimum  learner  control  may  dictate  a  strict 
presentation  sequence  for  a  course's  atoms,  while  a  strategy  that  provides  maximum  learner 
control  may  remove  all  constraints  upon  the  sequence  of  atom  presentation.  The  DDT's 
instructional  atom  database  and  its  course  outline  database  separate  the  system's  instructional 
content  from  instructional  strategy. 

Course  outlines  are  produced  by  an  outline  builder  and  are  executed  by  the  system's 
sequencer.  The  outline  builder  resembles  an  authoring  application,  but  instead  of  enabling  the 
production  of  instructional  atoms,  it  enables  production  of  course  outlines.  The  DDT's  sequencer 
is  responsible  for  invoking  presentation  applications  (and  thus  displaying  instructional  atoms)  in 
accordance  with  a  course  outline.  Information  regarding  a  learner's  performance  is  recorded  by 
the  sequencer  in  the  DDT's  performance  database.  The  system's  report  generator  organizes 
performance  data  for  display  and  printing. 

The  DDT's  final  database,  the  operator  database,  identifies  operators  and  classifies  them 
as  learners,  instructors,  authors,  or  researchers.  It  also  associates  each  operator  with  a  course  and 
a  strategy.  The  system's  administrator  process  manages  this  database  and  provides  the  system's 
login  services. 
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III.  Instructional  Strategy  Implementation 


This  section  describes  how  the  lessons  and  the  exercises  were  implemented.  The  starting 
point  for  the  implementation  was  an  exercise  scenario  developed  in  Phase  1,  with  the  aid  of  LC2 
subject  matter  experts  (SMEs).  We  begin  our  account  of  instructional  strategy  implementation 
with  the  lessons  and  then  turn  to  the  exercises. 

3.1.  Lesson  Implementation 

3.1.1.  Lesson  Design  and  Development 

Two  lessons  providing  the  essential  knowledge  required  for  playing  the  exercise  scenario 
were  developed.  The  process  involved  the  four  steps  of  topic  determination,  syllabus  layout, 
lesson  specification,  and  lesson  development.  Required  lesson  topics  were  determined  by 
considering  the  logical  prerequisites  for  a  basic  understanding  of  the  system  and  the  artificial 
world  used  in  the  exercise.  The  topics  were  assigned  to  lessons  and  sections  within  lessons,  with 
five  sections  in  the  first  and  seven  sections  in  the  second  lesson.  The  sequence  of  sections  was 
again  determined  on  the  basis  of  logical  prerequisite  considerations.  Detailed  specifications  for 
each  section  in  each  lesson  were  then  written.  These  specifications  contained  the  exact  text  and 
ideas  for:  (a)  illustrations;  and  (b)  instructional  elements  planned  for  each  section. 

While  these  design  steps  were  being  performed,  development  began  on  the  user  interface 
for  the  lessons.  This  interface  was  designed  to  permit  changes  in  the  sequence  of  instructional 
elements,  changes  in  the  amount  of  control  that  learners  would  have  over  that  sequence,  and  the 
systematic  inclusion  or  exclusion  of  particular  types  of  instructional  elements.  A  basic  version  of 
this  interface  was  developed  using  Toolbook  (Version  1.53)  software. 

With  the  lesson  interface  in  place,  the  specifications  were  transformed  into  working 
lessons.  The  lessons  were  then  functionally  refined  and  artistically  embellished.  The  end  product 
of  lesson  implementation  was  a  modular  structure  of  instructional  elements  connected  by  a 
modifiable  script  determining  their  sequencing  constraints.  The  current  script  represents  one 
possible  micro-strategy  for  the  lessons. 
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3.1.2. 


Taking  a  Lesson 


The  modular  structure,  the  user  interface,  and  the  "look  and  feel"  of  the  lessons  are 
illustrated  in  Figures  3-9.  This  sequence  follows  the  process  a  student  goes  through  when  taking 
a  lesson.  In  the  DDT  Beta  Version  0.1,  a  user  accesses  a  lesson  from  an  icon  in  a  program  group. 
The  lesson  opens  with  lesson  title  screen  such  as  the  one  shown  in  Figure  3. 

Figure  3.  Lesson  title  screen. 


After  clicking  on  the  "Begin"  button,  the  user  sees  a  Lesson  Map  (Figure  4).  The 
diagram  in  the  lesson  map  shows  the  sections  in  a  lesson.  The  hierarchical  structure  indicates 
prerequisite  relationships.  A  color  code  indicates  whether  a  section  is  accessible  and  its 
completion  status. 
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Figure  4.  The  lesson  map 


When  the  student  clicks  on  an  accessible  section,  the  Section  Title  (Figure  5). 
Figure  5.  Section  title  screen. 


After  clicking  on  the  "Begin"  button  of  the  Section  Title  Screen,  the  control  and  view 
screen  for  the  instructional  elements  in  a  section  appears  (Figure  6).  The  instructional  elements 
represent  the  lowest  level  of  modularity  in  the  lessons.  Accessibility  of  elements  is  indicated  by 
bright  buttons  on  the  control  panel. 

Figure  6.  The  control  panel. 


Clicking  on  a  bright  button  on  the  control  panel  displays  the  corresponding  element  either 
on  the  Main  Viewer  or  on  a  special  panel.  The  element  types  labeled  "ATTENTION", 
"ORGANIZER",  "RECALL",  "OBJECTIVE",  "SUMMARIZER",  and  "SYNTHESIS",  are 
usually  displayed  on  the  Main  Viewer  (Figure  7) 
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Figure  7.  The  main  viewer 


Element  types  labeled  "CONTENT”,  "DEMO",  "EXAMPLE",  "PRACTICE",  and 
"TEST"  are  displayed  on  special  panels,  such  as  the  practice  item  (Figure  8). 
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Figure  8.  Practice  and  test  elements  panels. 
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I  Next  Question 


Content  elements  often  include  several  "pages"  worth  of  information.  As  such,  the  first 
display  will  provide  an  overview  (Figure  9).  Lower  levels  of  information  can  then  be  accessed 
from  there  by  means  of  control  buttons  on  the  left. 


Figure  9.  Content  elements  panels. 
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3.2.  Exercise  Implementation 


3.2.1.  Exercise  Design  Issues 

As  previously  indicated,  a  number  of  design  issues  had  to  be  resolved  in  the  course  of 
developing  the  exercises.  The  two  most  significant  issues;  namely,  the  problems  of  simulating 
causes  and  simulating  time  are  discussed  below.  These  issues  were  significant  from  the 
viewpoint  of  both  software  design  and  instructional  design. 

3.2.1. 1.  The  Problem  of  Simulating  Causes 

Discussions  with  subject  matter  experts  (SME)  first  led  to  a  suspicion  and  later  to 
confirmation  that  solutions  to  logistical  problems  depend  to  a  very  large  degree  on  the  cause  or 
causes  of  the  problem.  For  example,  an  operational  unit  may  suddenly  be  faced  with  a  severe 
shortage  of  food  supplies.  If  this  problem  is  caused  by  local  enemy  action,  then  the  necessary 
and  sufficient  solution  is  speedy  resupply.  If  this  problem  is  caused  by  discovery  of 
contamination,  speedy  resupply  is  still  a  necessary  solution,  but  certainly  not  a  sufficient  one.  In 
this  case  it  is  likely  the  problem  is  more  global  than  local.  Any  particular  shortage  can  have  a 
number  of  different  causes. 

It  is  relatively  easy  to  generate  a  shortage  in  a  simulation,  but  it  is  quite  a  different  task  to 
simulate  the  array  of  potential  causes  and  generate  throughout  the  system,  information  traces  that 
indicate  a  particular  cause.  At  the  same  time,  it  is  essential  that  a  decision  training  system  be 
capable  of  presenting  widely  varying  problem  sets  that  representative  of  the  variability  in  the 
decision  domain.  The  challenge  of  generating  appropriate  problems  is  further  complicated  by  the 
need  to  control  the  complexity  and  uncertainty  levels  of  practice  problems  in  training  systems. 

3.2.1.2.  The  Case  Approach 

The  problem  of  simulating  causes  led  to  a  significant  shift  in  the  design  of  the  simulation. 
The  Phase  1  design  was  based  on  the  assumption  that  the  simulation  could  generate  the  practice 
problems  based  on  outputs  obtained  from  a  relatively  simple  conflict  model.  Once  we  were 
certain  that  problem  causes  were  indeed  significant  decision  determinants,  we  were  also  certain 
that  the  "automated  case  generation"  paradigm  was  not  feasible.  The  solution  was  to  shift  to 
authored  cases. 
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As  illustrated  in  Figure  10,  a  case  consists  of  two  main  elements:  (1)  the  causal  story,  and 
(2)  logistics  simulation  changes.  The  story  provides  explanatory  information  about  why  certain 
logistics  events  (e.g.,  shortages,  consumption  increases,  etc.)  have  occurred.  The  details  of  the 
story  influence  how  the  student  is  expected  to  react  to  the  problem.  The  elements  that  make  up 
the  story  are  structured  to  allow  the  student  to  "discover"  the  story  and  to  permit  evaluation  of 
the  student's  discovery  process.  Logistics  simulation  changes  are  a  description  of  the  changes 
that  need  to  be  made  to  the  logistics  simulation  to  "cause"  the  logistics  event  associated  with  the 
case. 


The  causal  story  (Figure  1 1)  consists  of  messages  and  information  deposits.  Messages, 
like  mail,  are  unsolicited  information.  They  arrive  at  the  message  desk  during  the  course  of  a 
decision  making  episode  (DME);  the  student  can:  (a)  open  the  message  and  view  their  contents; 
(b)  file  for  later  action;  or  (c)  dispose  of  them. 

Figure  10.  DDT's  case  approach. 
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Information  deposits  are  solicited  communications.  The  student  must  explicitly  query  an 
agency  to  get  the  information  contained  in  an  information  deposit.  Each  information  deposit  is 
associated  with  a  responding  agency  and  an  eliciting  topic  and  contains  information  that  is 
presented  when  the  student  asks  the  agency  about  that  topic.  A  topic  is  a  word  or  a  short  phrase 
that  can  serve  as  the  object  of  a  student's  query  of  an  agency.  This  word  or  phrase  will  have 
appeared  in  some  information  previously  viewed  by  the  student.  As  illustrated  in  Figure  11, 
messages  and  information  deposits  communicate  their  information  using  an  information  packet. 
An  information  packet  is  the  smallest  element  of  information  and  contains  the  actual  text  or 
pictures  that  present  the  information  to  the  student. 
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Logistics  simulation  changes  are  a  description  of  the  changes  that  need  to  be  made  to  the 
logistics  simulation  to  "cause"  the  logistics  event  associated  with  a  case.  There  are  five  different 
types  of  logistics  simulations  changes:  (a)  adding/removing  resources;  (b)  tagging  resources;  (c) 
adjusting  consumption  rates;  and  (d)  adjusting  supply  rates. 

Adding/Removing  Resources.  Resources  can  be  added  to  the  simulation.  Adding 
resources  might  be  necessary  to  give  the  student  specific  options  for  correcting  a  shortage. 
Resources  can  also  be  removed  from  the  simulation.  If  commodities  are  removed,  that  will  cause 
a  possible  one-time  shortage.  If  transportation  resources  are  removed,  that  will  cause  a  possible 
supply  rate  problem. 

Tagging  Resources.  Resources  can  be  tagged  with  information  that  relates  both  to  the 
story  and  to  the  resources'  behavior  in  the  simulation.  For  example,  food  kits  might  be  marked  as 
contaminated  and  not  available  for  consumption.  This  would  cause  a  shortage  of  food  kits.  The 
food  kits,  however,  would  appear  normal  if  the  student  formulated  a  query  about  food  kits.  Only 
when  the  student  asked  about  an  appropriate  topic  would  the  information  about  the  contaminated 
food  kits  be  revealed. 

Adjusting  Consumption  Rates.  Consumption  rates  can  be  adjusted.  An  increase  in 
consumption  at  a  particular  agency  will  cause  a  shortfall  if  the  student  does  not  adjust  the 
appropriate  shipment  schedules. 

Adjusting  Supply  Rates.  Supply  rates  can  be  changed.  An  decrease  in  supply  from  a 
particular  agency  will  cause  a  shortfall  if  the  student  does  not  adjust  other  supply  rates  or  adjust 
shipment  schedules. 

3.2.I.3.  The  Problem  of  Simulating  Time 

A  second  problem  was  presented  by  the  duration  of  logistical  decision  making  tasks  in 
the  real  world.  The  time  from  task  recognition  to  decision  implementation  may  be  very  short; 
however,  the  time  from  implementation  to  the  appearance  of  results  is  usually  measured  in  days. 
Results  from  a  logistics  action  represent  "natural"  feedback  information  to  the  decision  maker. 
This  natural  feedback  is  extremely  desirable  and  important  for  training.  However,  it  would  be 
impractical  to  have  a  student  wait  for  days  for  feedback  to  maintain  time  profile  correspondence 
between  real  and  training  tasks.  During  training,  the  student  should  receive  feedback  on  the 
results  of  their  decisions  within  minutes.  The  challenge  then,  was  to  provide  natural  feedback 
within  minutes  while  clearly  indicating  that  these  minutes  represent  days  on  the  real  time  scale. 
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3.2. 1.4.  The  DME  Approach 

The  solution  to  simulating  time  in  DDT  Beta  Version  0.1  was  based  on  the  notion  of 
"pulling  shifts"  in  a  Logistics  Readiness  Center  (LRC)  (see  Figure  12).  Each  shift  is  represented 
by  a  Decision  Making  Episode  (DME)  in  the  exercise. 


Figure  12.  Decision-making  episodes. 


The  student  attends  a  series  of  shifts  that  are  days  apart  in  scenario  but  only  minutes  apart 
as  DME’s  in  actual  training  time.  The  student  implements  decisions  in  one  DME  and  the 
simulation  proceeds  in  "fast  forward"  mode  to  the  next  DME.  The  next  DME  occurs  a  certain 
number  of  days  later,  but  only  minutes  of  real  time  will  have  passed.  The  time  interval  between 
shifts  was  set  at  five  days  to  allow  sufficient  time  for  supplies  to  travel  from  source  to  destination 
(i.e.  to  allow  sufficient  time  for  the  appearance  of  decision  results).  These  results  can  currently 
be  accessed  by  the  student  by  means  of  a  query  facility.  In  subsequent  versions  of  the  system, 
the  student  will  receive  an  update  briefing  at  the  beginning  of  each  new  DME/shift. 


3.2.2.  Making  Decisions  in  an  Exercise 


Exercises  provide  decision  making  practice  in  an  artificial  simulation  environment  called 
STARWORLD.  Each  exercise  rolls  off  in  three  phases.  The  first  phase,  called  Orientation 
Phase  or  Briefings,  provides  the  student  with  a  opportunity  to  become  familiar  with  the  starting 
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situation  for  the  exercise  scenario.  The  "war"  starts  and  the  clock  begins  to  tick  when  the  student 
enters  the  second  phase,  called  the  Operations  Phase.  During  this  phase  the  student  does  duty 
over  several  shifts  in  an  LC2  node  that  represents  an  LRC.  During  each  shift  (or  DME)  the 
student  is  exposed  to  several  decision-making  problems.  After  the  last  shift  of  an  exercise,  the 
student  enters  the  third  phase  of  an  exercise:  the  Debriefing  Phase.  During  the  final  phase,  the 
student  has  an  opportunity  to  review  a  number  of  performance  scores  and  the  final  supply  status 
at  the  operational  unit(s)  for  which  they  were  responsible.  Each  of  these  three  phases  is 
described  in  more  detail  below. 

3.2.2.I.  The  Orientation  Phase 

Familiarization  with  the  starting  situation  of  an  exercise  is  enabled  by  three  Information 
Sources:  Briefings,  Orders,  and  References  (Figure  13). 


Figure  13.  The  Orientation  Phase 


Information  Sources 
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Briefings  and  orders  are  structured  to  resemble  typical  military  situation  briefings  and 
daily  orders.  They  characterize  the  general  military  situation  and  provide  several  special 
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sections,  one  of  which  deals  with  logistics.  The  Briefings  and/or  Orders  refer  to  objects  in  the 
artificial  STARWORLD  in  which  the  scenario  takes  place.  The  student  may  want  to  refresh  their 
memory  about  one  or  more  of  these  objects  before  proceeding  with  the  Operations  phase.  To  do 
so,  they  will  be  able  to  access  the  References  in  future  implementations  of  the  system. 

3.2.2.2.  The  Operations  Phase 

This  phase  provides  the  actual  practice  opportunities  for  decision  making.  As  indicated 
before,  such  opportunities  are  provided  during  DME’s  that  represent  shifts  in  an  LRC.  One  or 
more  problems  (also  called  "cases")  can  be  scheduled  for  each  DME.  The  subsequent  paragraphs 
describe  how  decision  making  practice  is  implemented  in  the  current  system  version  by 
"walking"  through  a  case  from  beginning  to  end.  The  description  is  organized  in  terms  of  the 
Timeline  Model  developed  during  Phase  1  of  this  project  (Figure  14).  The  model  is  subdivided 
into  four  parts  or  processes:  Recognition,  Uncertainty  Reduction  and  Option  Editing, 
Implementation,  and  Feedback. 

Figure  14.  The  timeline  model. 
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A  major  consideration  in  the  implementation  of  decision  making  practice  was  fidelity  to 
the  actual  decision  making  tasks  faced  by  personnel  in  LC2.  At  the  same  time  the 
implementation  had  to  be  controllable  in  terms  of  the  theoretical  constructs  developed  during 
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Phase  1  of  the  project.  Of  particular  concern  in  this  respect  was  the  ability  to  control  the  type 
and  relative  amount  of  uncertainty  presented  by  any  particular  case. 


Recognition.  During  the  recognition,  process  the  student's  task  is  to  recognize  the 
existence  of  a  decision  problem  to  be  dealt  with.  Recognition  is  based  on  information  that  is 
either  supplied  by  the  system  or  solicited  by  the  student.  Information  supplied  by  the  system 
arrives  in  the  form  of  one  or  more  messages  on  the  Message  Desk.  These  messages  are  either 
genuine  alerts  to  problems  the  student  is  expected  to  recognize  or  they  are  distracter  messages. 
Either  type  of  message  can  be  ambiguously  worded  thus  making  it  difficult  to  distinguish 
genuine  alerts  from  distracters.  Figure  15  illustrates  an  example  of  how  such  messages  are 
presented  and  worded. 

Figure  15.  Sample  message. 


To  clarify  whether  a  message  hints  at  a  real  problem  or  not,  the  student  must  reduce 
situation  uncertainty.  This  is  accomplished  by  soliciting  information  from  the  various  agencies 
participating  in  the  scenario.  These  agencies  either  know  something  about  a  topic  mentioned  in  a 
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previously  seen  message  or  not.  If  they  do,  they  provide  additional,  more  or  less  clarifying, 
information.  Figure  16  illustrates  the  result  of  querying  an  agency  that  has  further  information 
about  a  topic. 


Figure  16.  Sample  information  from  agencies. 


Operations  ;  :  -  Day  1.  Monday  12:0B  : 


The  student's  uncertainty  reduction  efforts  are  more  or  less  efficient  depending  on  how 
many  queries  are  made  to  recognize  the  existence  of  a  genuine  problem.  Efforts  are  effective  to 
the  degree  that  the  student  discovers  all  the  pieces  of  information  deposited  with  agencies  that 
are  expected  to  know  something  about  the  problem  presented  in  the  case. 


Once  the  student  discovers  there  is  a  problem  to  be  dealt  with,  they  can  put  it  on  his 
agenda  for  problem  solving.  For  this  purpose  the  system  provides  a  To-Do  List.  If  the  student 
discovers  that  a  message  is  a  distracter,  they  can  either  file  it  away  for  later  reference,  pass  it  on 
to  other  agencies,  or  throw  it  away.  To  deal  with  a  message  in  any  of  these  ways  the  student 
simply  drags  the  message  over  to  the  respective  icon  (see  Figure  17).  To  put  a  recognized 
problem  actually  on  the  To-Do  list,  the  To-Do  List  window,  shown  in  Figure  17  has  to  be 
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brought  up  by  a  double-click  on  the  icon.  In  that  window  a  mechanism  is  provided  to  enter  the 
problem  on  the  list  and  to  name  it.  If  the  list  already  contains  problem  names,  the  student  can 
adjust  their  order  and  thus  assign  relative  priorities. 

Figure  17.  The  to-do  list  window. 


Once  the  student  has  accomplished  all  that,  the  system  assumes  the  problem  is  recognized 
and  marks  the  minutes  of  playtime  that  have  passed  since  the  presentation  of  the  first  message 
hinting  at  the  existence  of  a  problem. 

Uncertainty  Reduction  and  Option  Editing.  Problems  on  the  To-Do  List  must  be  dealt 
with.  Dealing  with  a  problem  means,  in  the  most  general  sense,  correcting  some  deficiency  at 
some  operational  unit.  To  correct  a  deficiency,  the  student  must  set  a  goal  and  determine  a 
means  to  achieve  that  goal.  The  basic  means  to  achieve  goals  are  sourcing  and  transportation 
(i.e.,  the  student  must  find  appropriate  sources  for  the  needed  commodities  and  appropriate 
means  of  transporting  them  to  where  they  are  needed).  Since  many  different  combinations  of 
sourcing  and  transportation  are  possible,  the  student  must  reduce  goal  uncertainty  and  option 
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uncertainty  and  then  reach  a  decision  to  implement  an  option.  Two  facilities  are  provided  for 
these  purposes:  the  Data  Queries  and  the  Decision  Options  window. 


The  Data  Queries  simulate  access  to  computer  data  bases.  The  current  implementation 
allows  the  student  access  to  four  types  of  queries:  Consumables  (type,  location,  owner,  number), 
Transport  Schedules  (what  missions  travel  between  any  two  points,  how  much  free  capacity  do 
they  have  on  average),  Shipments  (source  and  destination  for  any  consumable  shipment,  next 
shipment  departure),  and  Shipment  Schedules  (source  and  destination  for  any  consumable 
shipment,  next  shipment  departure,  frequency  of  shipments).  The  student  can  thus  reduce 
uncertainty  with  respect  to  sourcing  and  transportation  by  means  of  the  data  query  facilities.  One 
example  (Consumables)  of  a  Data  Query  window  is  shown  in  Figure  18. 

Figure  18.  Data  query  window. 
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The  Decision  Options  window  pictured  in  Figure  19  allows  the  student  to  set  a  goal,  pick 
an  option  class  from  a  menu,  and  define  specific  options.  Goals  are  articulated  as  sentences.  The 
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mechanism  is  a  series  of  drop-down  menus.  Up  to  two  goals  can  be  specified  in  the  current 
version. 

Figure  19.  The  decision  options  window. 
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Once  a  goal  is  specified,  option  classes  can  be  selected  from  a  drop-down  menu  of  six 
classes  that  encompass  all  general  means  to  achieve  correction  of  a  deficiency.  Up  to  three 
option  classes  can  be  accommodated  in  the  window.  Congruence  between  a  goal  and  the  chosen 
option  class  is  not  evaluated  in  the  current  system  version.  Once  the  student  has  chosen  an 
option  class,  he/she  can  define  up  to  three  options  that  specify  in  detail  how  that  option  class  is  to 
be  instantiated.  The  current  system  version  does  not  check  congruence  between  specific  options 
and  option  class. 

Specific  options  are  articulated  as  new  shipments,  changes  to  shipment  schedules,  new 
transportation  missions,  or  changes  to  transportation  missions.  Any  specific  option  can  contain 
one  or  more  of  any  of  these  elements.  Figure  20  illustrates  a  fully  specified  new  shipment.  This 
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shipment  can,  and  in  this  case  actually  does,  represent  the  only  element  in  an  option  (i.e.,  it  is  a 
full  specification  of  the  option). 

Figure  20.  Shipment  option  window. 


Figtrter  Canisters  j±  jjStar  Depot  SLC  52 


Fuels  Management  SB1 0 


Unadilla 


SB  Warner  Robins 


RunRoverl:  Unadilla,  Unadilla,  Rover 


FlyShortyl :  Warner  Robins,  Warner  Robins,  Shorty 


Herpes 


Warpl:  Luna,  Luna,  Milofant 


Herpes 


FlyShorty2:  Herpes,  Herpes,  Shorty 


StarBaselO 


Cancel 


Shipment 


Source 


Commodity 


Recipient 


Freight  Mission 


iWarner  Robins 

.  ;  .  :  '  •  •..  ! 

s 

Implementation.  The  student  can  work  on  goals,  option  classes  and  specific  options  as 
little  or  as  much  as  he/she  wants.  For  implementation,  a  minimum  of  one  articulated  goal,  one 
option  class,  and  one  fully  specified  option  is  required.  The  act  of  implementation  is  very  simple: 
The  option  the  student  has  decided  on  is  selected  and  an  "Implement"  button  is  clicked.  If  the 
option  is  physically  feasible,  it  is  implemented  by  the  system;  otherwise,  error  messages  direct 
the  student  to  correct  incomplete  or  incorrect  specifications. 

Feedback.  The  system  currently  does  not  provide  immediate,  "artificial"  feedback  on 
implemented  or  contemplated  options.  Natural  feedback  is  available  by  accessing  data  queries 
during  the  next  DME.  The  data  queries  produce  tabular  displays  that  show  how  consumables 
have  moved  through  the  scenario  since  the  last  DME.  These  displays  are  not  easy  to  decipher. 
Natural  feedback  will  therefore  take  the  form  of  Update  Briefings  in  subsequent  system  versions. 
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3.2.23.  The  Debriefing  Phase 

The  Debriefing  Phase  provides  delayed  artificial  and  natural  feedback  on  the  entire 
exercise  as  depicted  in  Figure  21. 


Figure  21.  The  debriefing. 
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Artificial  feedback  consists  of  a  variety  of  performance  scores  that  summarize 
performance  for  each  DME  (see  upper  half  of  Figure  21).  Performance  for  individual  decision 
problems  can  be  seen  only  if  but  one  problem  was  scheduled  for  a  DME.  Below  these 
performance  scores  is  a  table  showing  the  final  supply  status  at  the  units  the  student  had  to  take 
care  of.  This  status  information  is  seen  as  natural  feedback,  since  it  would  be  available 
"naturally"  in  an  exercise  or  operational  situation. 
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IV.  SUMMARY 


During  Phase  2,  a  decision-making  training  prototype  was  developed.  This  prototype  is 
more  than  a  mere  demonstration  of  the  user  interface.  It  is  a  fully  functional,  albeit  not 
exhaustively  tested,  prototype  of  the  core  components  of  the  training  system  this  project  aims  to 
develop.  The  prototype  consists  of  two  full  lessons  and  one  simulation  exercise.  The  latter 
consists  of  a  simulation  engine,  a  user  interface,  and  a  scenario,  and  is  implemented  with  more 
than  40,000  lines  of  code.  The  two  Toolbook  lessons  consist  of  a  total  of  243  pages.  The  full 
prototype  system  occupies  four  high-density  disks.  The  system  thus  represents  a  significant 
amount  of  work  in  quantitative  terms. 

The  quality  of  the  work  that  has  been  accomplished  must  be  viewed  from  two  points  of 
view.  One  point  of  view  is  the  graphic  look  and  feel.  Considerable  care  has  been  taken  to  give 
the  system  a  finished,  professional,  and  appealing  look.  As  successful  as  this  effort  has  been,  this 
quality  issue  is  the  less  important  of  the  two.  More  significant  is  the  quality  of  the  system  in 
terms  of  its  congruence  with  the  theoretical  background  and  its  fidelity  to  the  decision  making 
tasks  that  LC2  personnel  have  to  perform  in  the  real  world.  It  is  fair  to  say  that  the  system's 
congruence  with  theory  is  nearly  exact.  However,  it  is  equally  fair  to  say  that  task  fidelity  is  at 
this  point  not  yet  where  it  ought  to  be.  It  is  clear  that  the  achievement  of  the  desired  level  of  task 
fidelity  would  require  an  evolutionary  process  that  is  fed  by  feedback  from  SME’s  who  have  the 
opportunity  to  work  with  the  prototype.  This  is  the  primary  reason  why  the  development  of  the 
prototype  has  been  pushed  as  far  as  it  has  been  during  Phase  2.  We  now  have  two  years  before 
the  scheduled  end  of  the  project,  the  vehicle  to  solicit  SME  feedback  and  thus  to  engage  in  that 
planned  evolutionary  process. 

While  the  key  instructional  and  software  implementation  issues  have  been  solved  at  this 
point,  much  remains  to  be  done.  Over  the  past  year,  particularly  during  the  final  months  when 
the  system  began  to  be  working  as  a  whole,  we  have  discovered  many  opportunities  for  the 
improvement  of  the  system.  It  is  doubtful  that  many  of  these  ideas  and  opportunities  could  have 
been  discovered  without  taking  prototype  development  as  far  as  it  has  gone  in  this  phase.  As  this 
project  passes  its  midpoint  in  time  and  transitions  from  Phase  2  to  the  final  Phase  3,  it  is 
imperative  that  Phase  3  be  started  with  a  thoughtful  review  of  these  improvement  opportunities 
as  they  relate  to  the  available  resources  and  the  project's  ultimate  goals. 
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